ImpactMojo ImpactMojo
Premium

Econometrics 101

Advanced Causal Inference Methods
ImpactMojo Workshop Series • Instrumental Variables, DiD, RDD & Panel Methods
75-90 Minutes

Workshop 2: Identification Strategies for Policy Evaluation

Target Audience: Impact evaluators, policy researchers, applied economists, and data scientists conducting causal analysis

Prerequisites: Econometrics 101 Workshop 1, understanding of regression analysis and causal inference concepts

Materials Needed: Statistical software (R/Stata/Python), sample datasets with policy variation

Learning Objectives

By the end of this workshop, participants will be able to:

Part 1: Instrumental Variables (IV) Estimation

25 minutes

Classic Example: Returns to Education using Distance to School

The Endogeneity Problem: Education is correlated with unobserved ability, motivation, and family background

The Instrument: Distance from home to nearest school

  • Relevance: Farther distance → less likely to attend school
  • Exclusion: Distance affects income only through education (not directly)

Two-Stage Estimation:

  • First stage: Education = α₀ + α₁Distance + α₂X + v
  • Second stage: Income = β₀ + β₁Education̂ + β₂X + u

Key Finding: IV estimates often larger than OLS (10-15% vs 7-8% returns per year)

IV Identification Requirements

Three Key Assumptions:

1. Relevance (First Stage)

Cov(Z, X) ≠ 0

  • Instrument must predict treatment
  • Test: F-statistic > 10 in first stage
  • Weak instruments → biased estimates
2. Exclusion Restriction

Cov(Z, u) = 0

  • Instrument affects outcome only through treatment
  • Untestable - requires theoretical justification
  • Most critical and controversial assumption
3. Monotonicity

No defiers

  • Treatment response goes in same direction
  • Enables interpretation as LATE
  • Often plausible but not always testable
4. Independence

Instrument as-good-as-random

  • No confounders of Z and Y
  • Can often control for observables
  • Test balance on pre-treatment variables

IV Evaluation Exercise (10 minutes)

Policy Question: Does health insurance increase healthcare utilization?

Proposed Instruments - Evaluate each:

Instrument Relevance? Exclusion Restriction? Valid?
Employer offers insurance Strong predictor Employers might select healthy workers Questionable
State mandate laws Affects coverage rates Policy exogenous to individuals Likely valid
Income level Affects insurance purchase Income directly affects healthcare demand Invalid
Lottery random numbers In lottery-based expansion Random assignment Gold standard

Part 2: Difference-in-Differences (DiD)

25 minutes

The DiD Logic: Policy Changes as Natural Experiments

Core Idea: Compare changes over time between treated and control groups

DiD Estimator = (Y₁ᵀ - Y₀ᵀ) - (Y₁ᶜ - Y₀ᶜ)

Two Sources of Variation:

  • Time variation: Before vs after policy implementation
  • Cross-sectional variation: Treated vs control units

Key Advantage: Controls for time-invariant unobserved differences between groups

Famous Example: Minimum Wage and Employment (Card & Krueger, 1994)

Setting: New Jersey raised minimum wage from $4.25 to $5.05; Pennsylvania unchanged

Before (Feb 1992) After (Nov 1992) Difference
New Jersey (Treated) 20.4 employees 21.0 employees +0.6
Pennsylvania (Control) 23.3 employees 21.2 employees -2.1
DiD Estimate Treatment Effect +2.7 employees

Interpretation: Minimum wage increase led to 2.7 more employees per restaurant

Surprising result! Contradicted standard economic theory predictions

DiD Regression Specification

Y_{it} = β₀ + β₁Treated_i + β₂Post_t + β₃(Treated_i × Post_t) + X_{it} + ε_{it}

Interpreting DiD Coefficients

  • β₁: Baseline difference between treated and control groups
  • β₂: Time trend affecting all units equally
  • β₃: DiD estimate - the causal effect of treatment

Key Assumption: Parallel Trends

Without treatment, treated and control groups would have evolved similarly

DiD Design Exercise (10 minutes)

Scenario: Evaluating the impact of a rural employment guarantee program (like MGNREGA)

Setting: Program rolled out to different districts in different phases

DiD Strengths
  • Controls for district-specific factors
  • Controls for time trends
  • Uses policy variation for identification
  • Can examine dynamics over time
DiD Concerns
  • Parallel trends assumption
  • Program placement might be endogenous
  • Spillover effects to control districts
  • Composition changes over time
Validity Tests
  • Pre-treatment trend analysis
  • Placebo tests on unaffected outcomes
  • Event study design
  • Robustness to different time windows
Expected Results
  • Increased rural employment
  • Higher agricultural wages
  • Reduced seasonal migration
  • Potential effects on women's work

Part 3: Regression Discontinuity Design (RDD)

20 minutes

RDD: Exploiting Arbitrary Thresholds

Core Insight: When treatment assignment based on crossing a threshold, units just above and below are very similar

Y_i = α + βT_i + f(X_i - c) + ε_i

Where:

  • X_i: Running variable (test score, income, etc.)
  • c: Cutoff threshold
  • T_i: Treatment indicator (1 if X_i ≥ c)
  • β: Local average treatment effect at cutoff

Indian Example: Merit-Based Scholarship Effects

Setting: Scholarship awarded to students scoring ≥75% on Class 10 exam

Research Question: Does receiving scholarship increase likelihood of completing Class 12?

RDD Logic:

  • Students scoring 74% vs 76% are very similar
  • Only difference: scholarship eligibility
  • Compare outcomes in narrow window around 75%
  • Difference = causal effect of scholarship

Key Assumption: No manipulation of test scores around cutoff

RDD Validity Checks

Essential Diagnostic Tests

1. Manipulation Test
  • McCrary density test
  • Smooth density around cutoff
  • No bunching of observations
  • Interview stakeholders about gaming
2. Covariate Balance
  • Pre-treatment characteristics smooth
  • No discontinuities in baseline variables
  • Age, gender, family background
  • Previous academic performance
3. Bandwidth Sensitivity
  • Results robust to different windows
  • Optimal bandwidth selection
  • Local linear vs polynomial fits
  • Cross-validation methods
4. Placebo Tests
  • Fake cutoffs where no treatment
  • Outcomes unaffected by treatment
  • Pre-treatment periods
  • Different subsamples
Common RDD Pitfalls
  • Fuzzy discontinuity: Not everyone complies with treatment assignment
  • Multiple cutoffs: Other programs might have same threshold
  • Narrow external validity: Effect only at cutoff, not generalizable
  • Small sample sizes: Limited observations near cutoff

Part 4: Panel Data Methods

15 minutes

Fixed Effects: Controlling for Unobserved Heterogeneity

Within Estimator Logic

Panel Data Model:

Y_{it} = α + βX_{it} + μ_i + λ_t + ε_{it}

Where:

  • μ_i: Individual fixed effects (time-invariant unobservables)
  • λ_t: Time fixed effects (common shocks)
  • β: Identified from within-unit variation over time

Key Insight: Each unit serves as its own control

Individual Fixed Effects

Controls for: Person-specific time-invariant factors

  • Ability, motivation, preferences
  • Family background, genetics
  • Time-invariant location factors

Example: Effect of job training on wages

Firm Fixed Effects

Controls for: Company-specific factors

  • Management quality, culture
  • Technology, capital stock
  • Industry characteristics

Example: Effect of training on productivity

Region Fixed Effects

Controls for: Location-specific factors

  • Geographic characteristics
  • Local institutions, culture
  • Market access, infrastructure

Example: Effect of roads on development

⏰ Time Fixed Effects

Controls for: Common time trends

  • Macroeconomic shocks
  • Policy changes affecting all units
  • Technological progress

Example: Economic cycles, monsoon years

Part 5: Method Selection and Validation

10 minutes

Choosing the Right Method: Decision Framework

Data Structure Assignment Mechanism Recommended Method Key Assumption
Cross-section Valid instrument available Instrumental Variables Exclusion restriction
Panel data Policy change over time Difference-in-Differences Parallel trends
Cross-section Threshold-based assignment Regression Discontinuity No manipulation
Panel data Selection on unobservables Fixed Effects No time-varying confounders
Cross-section Selection on observables Matching/Regression Conditional independence
Method Validation Checklist
  • Theoretical justification: Is the identification assumption plausible?
  • Empirical tests: Do diagnostic tests support validity?
  • Robustness checks: Are results stable across specifications?
  • Placebo tests: No effects where there shouldn't be any?
  • External validity: Relevant for policy questions of interest?
  • Statistical power: Sufficient sample size for reliable inference?
  • Effect size: Economically meaningful magnitudes?
  • Heterogeneity: Effects consistent across subgroups?

The Credibility Revolution in Economics

Modern economics emphasizes credible identification over statistical significance. A well-identified small effect is more valuable than a poorly-identified large effect. The goal is to find variation that is "as good as random" for answering policy questions.

Hierarchy of Evidence:

  1. Randomized controlled trials
  2. Natural experiments with clear identification
  3. Quasi-experimental methods with credible assumptions
  4. Observational studies with extensive controls
  5. Correlational analysis (descriptive only)

Advanced Econometrics Resources

Methods-Specific Texts:

Specialized Resources:

Software and Implementation:

Data for Practice:

Next Steps in ImpactMojo:

This handout is part of the ImpactMojo 101 Knowledge Series
Licensed under CC BY-NC-ND 4.0 • Free to use with attribution • www.impactmojo.in

For implementation code, diagnostic tests, and advanced method tutorials, visit the ImpactMojo platform.